Many software systems go through extensive testing before they are released in an attempt to eliminate bugs and improve performance. Often, testing is done ad-hoc. A tester creates a suite of tests based on past experience and/or previously encountered bugs. To create the suite of tests, the tester will often write a testing application that exercises the program in a variety of ways. This testing application is written such that at certain points, messages are logged to indicate success or failure of a particular test.
As more programs are becoming distributed across multiple machines, testing the programs is becoming more difficult. Furthermore, it may not be practical or feasible to manually diagnose problems based on logged messages collected from multiple machines. Very often, manual diagnosis requires significant time spent by experts who must have a detailed understanding of the inner workings of the program.